Pose Estimation and Body Tracking Using an Artificial Neural Network

ABSTRACT

According to one implementation, a pose estimation and body tracking system includes a computing platform having a hardware processor and a system memory storing a software code including a tracking module trained to track motions. The software code receives a series of images of motion by a subject, and for each image, uses the tracking module to determine locations corresponding respectively to two-dimensional (2D) skeletal landmarks of the subject based on constraints imposed by features of a hierarchical skeleton model intersecting at each 2D skeletal landmark. The software code further uses the tracking module to infer joint angles of the subject based on the locations and determine a three-dimensional (3D) pose of the subject based on the locations and the joint angles, resulting in a series of 3D poses. The software code outputs a tracking image corresponding to the motion by the subject based on the series of 3D poses.

BACKGROUND

Augmented Reality (AR) and virtual reality (VR) experiences mergevirtual objects or characters with real-world features in a way thatcan, in principle, provide a deeply immersive and powerfully interactiveexperience. Nevertheless, despite the relative success of digitalenhancement techniques in augmenting many inanimate objects, digitalaugmentation of the human body continues to present substantialtechnical obstacles. For example, due to the ambiguities associated withdepth projection, as well as the variations in human body shapes,three-dimensional (3D) human pose estimation remains a significantchallenge.

In addition to AR and VR applications, accurate body tracking, inparticular hand tracking, is important for effective use of the humanhand as a Human Computer Interface (HCl). Applications for which use ofthe human hand as an HCl may be advantageous or desirable include handtracking based character animation, for example. However, the challengesassociated with pose estimation present significant problems for handtracking as well. Consequently, there is a need in the art for a fastand accurate pose estimation and body tracking solution.

SUMMARY

There are provided systems and methods for performing pose estimationand body tracking using an artificial neural network, substantially asshown in and/or described in connection with at least one of thefigures, and as set forth more completely in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a diagram of an exemplary system for performing poseestimation and body tracking using an artificial neural network (ANN),according to one implementation;

FIG. 2 shows a diagram of an exemplary use case for the system of FIG. 1in which hand tracking is performed, according to one implementation;

FIG. 3 shows an exemplary diagram of a software code including atracking module having an ANN trained to perform pose estimation andsuitable for execution by a hardware processor of the system shown byFIGS. 1 and 2, according to one implementation;

FIG. 4 shows an exemplary diagram of a landmark detector of the trackingmodule shown in FIG. 3;

FIG. 5 shows an exemplary diagram of a joint angle encoder of thetracking module shown in FIG. 3;

FIG. 6 shows an exemplary diagram of an inverse kinematics ANN shown in

FIG. 3; and

FIG. 7 shows a flowchart presenting an exemplary method for performingpose estimation and body tracking using an ANN of the tracking module ofFIG. 3, according to one implementation.

DETAILED DESCRIPTION

The following description contains specific information pertaining toimplementations in the present disclosure. One skilled in the art willrecognize that the present disclosure may be implemented in a mannerdifferent from that specifically discussed herein. The drawings in thepresent application and their accompanying detailed description aredirected to merely exemplary implementations. Unless noted otherwise,like or corresponding elements among the figures may be indicated bylike or corresponding reference numerals. Moreover, the drawings andillustrations in the present application are generally not to scale, andare not intended to correspond to actual relative dimensions.

The present application discloses systems and methods for performingpose estimation and body tracking using one or more artificial neuralnetworks (ANNs) and in a manner that overcomes the drawbacks anddeficiencies in the conventional art. It is noted that, as defined inthe present application, an artificial neural network (ANN), or simplyneural network (NN) is a type of machine learning framework in whichpatterns or learned representations of observed data are processed usinghighly connected computational layers that map the relationship betweeninputs and outputs. A “deep neural network”, in the context of deeplearning, may refer to a neural network that utilizes multiple hiddenlayers between input and output layers, which may allow for learningbased on features not explicitly defined in raw data. “Online deeplearning” may refer to a type of deep learning in which machine learningmodels are updated using incoming data streams, and are designed toprogressively improve its performance of a specific task as new data isreceived and/or adapt to new patterns of a dynamic system. As such,various forms of ANNs may be used to make predictions about new databased on past examples or “training data”. In various implementations,ANNs may be utilized to perform image processing or natural-languageprocessing.

It is further noted that, as defined in the present application, a“hierarchical skeleton” or “hierarchical skeleton model” refers to asystem for describing a collection of bones, and the joints connectingthose bones, according to a hierarchy in which the location ororientation of a bone or joint is dependent on the position(s) ororientation(s) of one or more other bones and joints. This is incontrast to non-hierarchical skeletons in which individual bones andjoints are treated as being independent of one another.

FIG. 1 shows a diagram of an exemplary system for performing poseestimation and body tracking using an ANN, according to oneimplementation. As shown in FIG. 1, pose estimation and body trackingsystem 100 includes computing platform 102 having hardware processor104, system memory 106 implemented as a non-transitory storage device,and display 108. According to the present exemplary implementation,system memory 106 stores software code 110. It is noted that hardwareprocessor 104 may be implemented as one or more processors for executingsoftware code 110, such as one or more central processing units (CPUs)and/or one or more graphics processing units (GPUs), for example.

As further shown in FIG. 1, pose estimation and body tracking system 100is implemented within a use environment including cameras 120 a and 120b, and subject 124, which may be a human subject or robot, for example,having body 125 and hands 126 a and 126 b. Also shown in FIG. 1 areimages 122 a and 122 b of motion by subject 124, as well as trackingimage 156 corresponding to the motion by subject 124.

It is noted that in some implementations, pose estimation and bodytracking system 100 may be configured to perform hand tracking ofsubject 124, i.e., tracking of hand motions by subject 124. However, inother implementations, body 125 of subject 124 may be in motion relativeto cameras 120 a and 120 b. In those latter implementations, poseestimation and body tracking system 100 may be configured to track themotion through space of body 125 of subject 124, in addition to, or asan alternative to performing hand tracking of one or both of hands 126 aand 126 b of subject 124.

It is also noted that, although the present application refers tosoftware code 110 as being stored in system memory 106 for conceptualclarity, more generally, system memory 106 may take the form of anycomputer-readable non-transitory storage medium. The expression“computer-readable non-transitory storage medium,” as used in thepresent application, refers to any medium, excluding a carrier wave orother transitory signal that provides instructions to hardware processor104 of computing platform 102. Thus, a computer-readable non-transitorymedium may correspond to various types of media, such as volatile mediaand non-volatile media, for example. Volatile media may include dynamicmemory, such as dynamic random access memory (dynamic RAM), whilenon-volatile memory may include optical, magnetic, or electrostaticstorage devices. Common forms of computer-readable non-transitory mediainclude, for example, optical discs, RAM, programmable read-only memory(PROM), erasable PROM (EPROM), and FLASH memory.

Although computing platform 102 is shown as a desktop computer in FIG.1, that representation is also provided merely as an example. Moregenerally, computing platform 102 may be any suitable mobile orstationary computing device or system that implements data processingcapabilities sufficient to implement the functionality ascribed tocomputing platform 102 herein. For example, in other implementations,computing platform 102 may take the form of a laptop computer, tabletcomputer, or smartphone, for example.

In some implementations, subject 124 may be a user of computing platform102, and may interact with software code 110 to produce tracking image156 corresponding to motion by subject 124. For example, subject 124 maybe an animator or performance actor, motion-capture actor, etc.,situated in front of cameras 120 a and 120 b while moving one or both ofhands 126 a and 126 b, and may have those hand motions applied to ananimated character. Alternatively, subject 124 may use hands 126 a and126 b to grab the character, pose it as though it were a physicalcharacter, and have that pose applied to the animated character.

According to various implementations, tracking image 156, when generatedusing software code 110 executed by hardware processor 104, may bestored in system memory 106 and/or may be copied to non-volatilestorage. Alternatively, or in addition, in some implementations,tracking image 156 may be rendered on display 108 of pose estimation andbody tracking system 100. Display 108 may be implemented as a liquidcrystal display (LCD), a light-emitting diode (LED) display, an organiclight-emitting diode (OLED) display, or another suitable display screenthat performs a physical transformation of signals to light.

FIG. 2 shows a diagram of an exemplary use case for the system of FIG. 1in which hand tracking is performed, according to one implementation.FIG. 2 includes pose estimation and body tracking system 200, cameras220 a and 220 b, and hand 226 including exemplary skeletal landmarks 228a and 228 b in the form of joint positions, and exemplary joint angles238 a and 238 b. It is noted, that although joint positions 228 a and228 b and exemplary joint angles 238 a and 238 b correspond to jointinformation for the thumb and pinky fingers as shown, the processing ofskeletal landmarks, joint position, and joint angles, according tovarious implementations, may be readily applicable to the other skeletalfeatures of the hand 226, including those of the remaining fingers,palm, wrist, etc. Also shown in FIG. 2 are images 222 a and 222 b ofmotion by hand 226, wireless communication link 250 a coupling camera220 a to pose estimation and body tracking system 200, and wiredcommunication link 250 b coupling camera 220 b to pose estimation andbody tracking system 200.

Pose estimation and body tracking system 200 corresponds in general topose estimation and body tracking system 100, in FIG. 1, and thosecorresponding elements may share any of the features or functionalityattributed to either corresponding element by the present disclosure.That is to say, although not shown in FIG. 2, pose estimation and bodytracking system 200 may include features corresponding respectively tocomputing platform 102 including hardware processor 104, system memory106 storing software code 110, and display 108. Moreover, like poseestimation and body tracking system 200, pose estimation and bodytracking system 100 may be in wireless or wired communication withcameras 120 a and 120 b via links corresponding respectively to wirelesscommunication link 250 a and wired communication link 250 b.

In addition, hand 226, in FIG. 2, corresponds in general to either orboth of hands 126 a and 126 b of subject 124, in FIG. 1. As a result,hands 126 a and 126 b may share any of the characteristics attributed tohand 226 by the present disclosure, and vice versa. For example, likehand 226, hands 126 a and 126 b may share features correspondingrespectively to exemplary joint positions or other skeletal landmarks228 a and 228 b, and exemplary joint angles 238 a and 238 b.

Cameras 220 a and 220 b, and images 222 a and 222 b, in FIG. 2,correspond respectively in general to cameras 120 a and 120 b, andimages 122 a and 122 b, in FIG. 1. Consequently, cameras 120 a and 120b, and images 122 a and 122 b, may share any of the characteristicsattributed to cameras 220 a and 220 b, and images 222 a and 222 b, bythe present disclosure, and vice versa. It is noted that although FIGS.1 and 2 show two cameras 120 a/220 a and 120 b/220 b, thatrepresentation is merely exemplary. In other implementations, poseestimation and body tracking system 100/200 may use as few as onecamera, i.e., camera 120 a/220 a or camera 120 b/220 b, or may use more,or many more than two cameras.

It is further noted that in some implementations, pose estimation andbody tracking system 100/200 may be in communication with one or more ofcameras 120 a/220 a and 120 b/220 b (hereinafter “camera(s) 120 a/220 aand 120 b/220 b”) but may not include camera(s) 120 a/220 a and 120b/220 b. However, in other implementations, camera(s) 120 a/220 a and120 b/220 b may be included as part of pose estimation and body trackingsystem 100/200. Moreover, although FIGS. 1 and 2 show camera(s) 120a/220 a and 120 b/220 b as discrete elements, physically separate fromcomputing platform 102 of pose estimation and body tracking system100/200, in some implementations camera(s) 120 a/220 a and 120 b/220 bmay be integrated with computing platform 102. For example, inimplementations in which computing platform 102 takes the form of atablet computer or smartphone, camera(s) 120 a/220 a and 120 b/220 b maybe a still or video camera integrated with the tablet computer orsmartphone.

As noted above, camera(s) 120 a/220 a and 120 b/220 b may be still imagecamera(s) or video camera(s), such as digital still image or digitalvideo cameras. In some implementations, camera(s) 120 a/220 a and 120b/220 b may be configured to capture color or black and white monoculardigital images as images 122 a/222 a and 122 b/222 b. In one suchimplementation, camera(s) 120 a/220 a and 120 b/220 b may bered-green-blue (RGB) color camera(s), for example. Alternatively, or inaddition, camera(s) 120 a/220 a and 120 b/220 b may be depth camera(s),such as RGB-D camera(s). In other implementations, camera(s) 120 a/220 aand 120 b/220 b may be infrared (IR) camera(s), or may correspond to anyother suitable optical sensor(s) for obtaining images 122 a/222 a and122 b/222 b of body 125 and/or hand or hands 126 a/126 b/226(hereinafter “hand(s) 126 a/126 b/226”) of subject 124.

FIG. 3 shows exemplary software code 310 suitable for execution byhardware processor 104 of pose estimation and body tracking system100/200, in FIGS. 1 and 2, according to one implementation. As shown inFIG. 3, software code 310 may include tracking module 340 havinglandmark detector 342, inverse kinematics ANN 344, joint angle encoder346, and decoder 348. In addition, FIG. 3 shows series of images 322,locations 352 of 2D skeletal landmarks determined by landmark detector342 of tracking module 340, joint angles 338 inferred by joint angleencoder 346 of tracking module 340, 3D poses 354 reconstructed usingtracking module 340, and tracking image 356 generated using trackingmodule 340.

As further shown in FIG. 3, software code 310 can include trainingmodule 332, as well as training database 334 storing body image dataset336 a and dataset 336 b of corresponding 3D poses with depthinformation. Software code 310 corresponds in general to software code110, in FIG. 1, and those corresponding features may share any of thecharacteristics attributed to either corresponding feature by thepresent disclosure. That is to say, like software code 310, softwarecode 110 may include a tracking module corresponding to tracking module340, as well as features corresponding respectively to training module332, and training database 334 storing body image dataset 336 a anddataset 336 b of corresponding 3D poses with depth information. However,it is noted that although FIG. 3 depicts training module 332 andtraining database 334 as being included in software code 110/310, thatrepresentation is merely exemplary. In other implementations, trainingmodule 332 and training database 334 may be stored remotely fromsoftware code 110/310 and may be utilized to train tracking module 340on a computing platform other than computing platform 102.

It is further noted that, in some implementations, body image dataset336 a stored in training database 334 may include millions ofrealistically rendered body images, such as hand images for example.Dataset 366 b stored in training database 344 may include 3D poses anddepth information corresponding to the millions of body images includedin body image dataset 336 a. Moreover, in some implementations, bodyimage dataset 336 a and dataset 336 b may be purely synthetic datasets.For example, in the exemplary use case of hand tracking, the purelysynthetic datasets may comprise of millions of 2D landmark to jointangle correspondences that are constructed to cover substantially allpractical poses of a human hand. This may require careful modeling ofjoint angles, careful modeling of correlations among joint angles, andcareful modeling of common hand gestures.

Series of images 322 corresponds in general to images 122 a/222 a and122 b/222 b, in FIGS. 1 and 2. Thus, series of images 322 may share anyof the characteristics attributed to corresponding images 122 a/222 aand 122 b/222 b by the present disclosure, and vice versa. In addition,tracking image 356, in FIG. 3, corresponds in general to tracking image156, in FIG. 1, and those corresponding features may share any of thecharacteristics attributed to either feature by the present disclosure.Moreover, locations 352 of 2D skeletal landmarks correspond in generalto skeletal landmarks 228 a and 228 b, in FIG. 2, while joint angles 338inferred by joint angle encoder 346 correspond in general to exemplaryjoint angles 238 a and 238 b.

FIG. 4 shows an exemplary diagram of a landmark detector of trackingmodule 340 in FIG. 3. As shown in FIG. 4, landmark detector 442 includesmulti-stage hourglass network 460 having individual hourglass stages461(1) to 461(N). In one implementation, for example, N may equal four.That is to say multi-stage hourglass network 460 may include fourhourglass stages 461(1), 461(2), 461(3), and 461(N=4).

Also shown in FIG. 4 are 2D mappings 462 generated by multi-stagehourglass network 460, as well as skeletal landmark extraction block464. In addition, FIG. 4 shows series of images 422 and L1 loss 466.Series of images 422 corresponds in general to images 122 a/222 a and122 b/222 b, in FIGS. 1 and 2, as well as to series of images 322 inFIG. 3. Thus, series of images 422 may share any of the characteristicsattributed to corresponding images 122 a/222 a, 122 b/222 b, and seriesof images 322 by the present disclosure, and vice versa.

Landmark detector 442, in FIG. 4, corresponds in general to landmarkdetector 342 of tracking module 340, in FIG. 3, and those correspondingfeatures may share any of the characteristics attributed to eitherfeature by the present disclosure. Thus, although not shown in FIG. 3,landmark detector 342 may include features corresponding to multi-stagehourglass network 460, 2D mappings 462 generated by multi-stagehourglass network 460, and skeletal landmark extraction block 464.

For each image of series of images 322/422, multi-stage hourglassnetwork 460 can be used to predict the respective locations of skeletallandmarks. For example, in some implementations in which hand trackingis being performed, multi-stage hourglass network 460 may be used topredict the locations of twenty-one landmarks in the hand. The locationsof the skeletal landmarks may be represented as 2D mappings 462 in theform of heatmaps in the image plane. Such a heatmap encodes theprobability of finding a skeletal landmark at a particular location inthe input image. Multi-stage hourglass network 460 may output one 2Dmapping for every skeletal landmark. Consequently, in the exemplary usecase in which hand tracking is performed using twenty-one skeletallandmarks, multi-stage hourglass network 460 generates twenty-one 2Dmappings 462 for each image of series of images 322/422.

It is noted that most conventional neural network architectures thatpredict heatmaps are trained with direct supervision on the predictedheatmaps. The ground truth heatmaps that are necessary for suchsupervision are typically generated by blurring the position of thelandmark by a Gaussian distribution with a user defined standarddeviation. In contrast to such approaches, multi-stage hourglass network460 is trained without explicit supervision on the heatmaps. Rather,multi-stage hourglass network 460 outputs a set of latent 2D mappings462 from which sub-pixel accurate skeletal landmark positions may beextracted by skeletal landmark extraction block 464 using a spatialsoft-argmax operation.

Moreover, additional constraints are imposed on the positions of theskeletal landmarks by jointly regressing the heatmaps of bones thatconnect pairs of skeletal landmarks. The heatmaps of these bones arealso unsupervised. The pairwise multiplication of the heatmaps of twobones generates the 2D mapping of the location of the skeletal landmarkat their intersection. The position of the skeletal landmark can bere-extracted from the result of the multiplication and is forced to lieat the same location as the ground truth.

FIG. 5 shows an exemplary diagram of a joint angle encoder suitable foruse in tracking module 340 in FIG. 3. Joint angle encoder 546 isconfigured to learn joint angle latent space 570. It is noted that jointangles 238 a/238 b/338 are represented in tracking module 340 as complexmathematical quantities known as quaternions. Also shown in FIG. 5 areL2 loss 572, normalizing layer 574, and quaternion loss 576. Joint angleencoder 546 corresponds in general to joint angle encoder 346, in FIG.3. That is to say, joint angle encoder 346 may share any of thecharacteristics attributed to joint angle encoder 546 by the presentdisclosure, and vice versa.

Once 2D skeletal landmarks are detected on each image of series ofimages 322/422 using landmark detector 342/442, joint angle encoder346/546 may be configured to infer joint angles that can deform a riggedskeleton into a desired pose. In one implementation, for example, jointangle encoder 346/546 may take the form of a fully convolutionalWasserstein autoencoder.

In the exemplary use case of hand tracking, and using a purely syntheticdataset consisting of over three million 2D skeletal landmark to jointangle correspondences, joint angle encoder 346/546 can be trained to mapmultiple joint angles, such as fifteen joint angles for example, to lowdimensional joint angle latent space 570, and reconstructs them fromthere. Because the movement of fingers is strongly related, it iscontemplated that joint angle encoder 346/546 can learn the correlationsbetween the various joint angles when it maps them onto joint anglelatent space 570.

As noted above, the joint angles are represented as quaternions whenproviding them as an input to joint angle encoder 346/546. To ensurethat joint angle encoder 346/546 always outputs valid quaternions, jointangle encoder 346/546 is trained with two losses. The predictions fromthe final layer of joint angle encoder 346/546 may be directlysupervised with a mean square loss (MSE loss L2) 572 using ground truthquaternions. Additionally, normalization layer 574 can be used tonormalize the activations of the final layer and further supervise themusing quaternion loss 576 measuring the difference between the rotationsrepresented by the two quaternions. It is noted that training jointangle encoder 346/546 with MSE loss 572 in addition to quaternion loss576 ensures that the direct predictions from joint angle encoder 346/546are already close to a quaternion and helps speed up convergence duringtraining.

FIG. 6 shows an exemplary diagram of an inverse kinematics ANN oftracking module 340 in FIG. 3. As shown in FIG. 6, exemplary inversekinematics ANN 644 includes fully connected layers 678, each of whichmay include five hundred and twelve features, for example. Also shown inFIG. 6 are input skeletal landmark locations 652, joint angle latentspace 670, L2 loss 672, normalizing layer 674, and quaternion loss 676,as well as decoder 648, which may be implemented as a fully pre-trainedWasserstein decoder. Inverse kinematics ANN 644 corresponds in generalto inverse kinematics ANN 344, in FIG. 3. That is to say, inversekinematics ANN 344 may share any of the characteristics attributed toinverse kinematics ANN 644 by the present disclosure, and vice versa.

In addition, input skeletal landmark locations 652 correspond in generalto locations 352, in FIG. 3, and those corresponding features may shareany of the characteristics attributed to either corresponding feature bythe present disclosure. Moreover joint angle latent space 670, L2 loss672, normalizing layer 674, and quaternion loss 676 correspondrespectively in general to joint angle latent space 570, L2 loss 572,normalizing layer 574, and quaternion loss 576, in FIG. 5.

It is noted that once latent space 570/670 of plausible joint angles hasbeen learnt by joint angle encoder 346/546, fully connected inversekinematics ANN 344/644 may be trained to regress to latent space570/670. Joint angles may be reconstructed using pre-trained decoder348/648, whose weights are fixed during the training of inversekinematics ANN 344/644. At evaluation time too, inverse kinematics ANN344/644 works together with decoder 348/648 to predict plausible jointangles given locations 352 of 2D skeletal landmarks.

The functionality of software code 110/310 and tracking module 340 willbe further described by reference to FIG. 7 in combination with FIGS. 1,2, and 3. FIG. 7 shows flowchart 780 presenting an exemplary method forperforming pose estimation and body tracking using an ANN, according toone implementation. With respect to the method outlined in FIG. 7, it isnoted that certain details and features have been left out of flowchart780 in order not to obscure the discussion of the inventive features inthe present application.

As a preliminary matter, it is noted that tracking module 340 is trainedto track motions prior to its use in performing the method outlined byflowchart 780. Tracking module 340 may be trained using software code110/310, executed by hardware processor 104, and using training module332 and training database 334. As discussed in greater detail above byreference to FIGS. 4, 5, and 6, training of tracking module 340 mayinclude providing individual body images from body image dataset 336 aas training inputs to landmark detector 342 of tracking module 340.Training of tracking module 340 may continue iteratively until 3D poses354 and joint angles 238 a/238 b/338 determined using tracking module340 converge to the 3D pose and depth information correspondingrespectively to the body images used for training and stored in dataset336 b.

Referring now to FIG. 7 in combination with FIG. 1 through FIG. 6,flowchart 780 begins with receiving series of images 322/422 of motionby subject 124 (action 781). Regarding individual images 122 a/222 a/122b/222 b included in series of images 322/422, it is noted that each ofimages 122 a/222 a/122 b/222 b may include multiple digital RGB, RGB-D,or IR frames, for example, obtained by camera(s) 120 a/220 a and 120b/220 b, and each capturing a different pose of subject 124 duringmotion by subject 124. Alternatively, series of images 322/422 mayinclude multiple frames taken from a video clip obtained by camera(s)120 a/220 a and 120 b/220 b.

For example, in one implementation, series of images 322/422 may includea sequence of single monocular images portraying motion by body 125and/or hand(s) 126 a/126 b/226 of subject 124. As noted above, in someimplementations, subject 124 may be a human subject or a robot.Moreover, in some of those implementations, the motion captured byseries of images 322/422 may be or include a hand motion by the humansubject or robot.

Series of images 322/422 may be received from camera(s) 120 a/220 a and120 b/220 b via wireless communication link 250 a and/or wiredcommunication link 250 b. Series of images 322/422 may be received bysoftware code 110/310, executed by hardware processor 104 of computingplatform 102.

Flowchart 780 continues with, for each image of series of images322/422, using tracking module 340 trained to track motions to determinelocations 352 each corresponding respectively to a 2D skeletal landmarkof subject 124 based on constraints imposed by features of ahierarchical skeleton model intersecting at each 2D skeletal landmark(action 782). It is noted that although FIG. 2 depicts skeletallandmarks 228 a and 228 b as joint positions on hand(s) 126 a/126 b/226of subject 124, that representation is merely exemplary. More generally,skeletal landmarks 228 a and 228 b may correspond to locations of anyrelevant joint or other structural or mechanical point of interests ofbody 125 of subject 124. Thus, in addition to, or as an alternative tohand joints, skeletal landmarks 228 a and 228 b may correspond to thelocations of hip joints, leg joints, foot joints, shoulder joints, andarm joints of subject 124, as well as head, neck, and spine joints ofsubject 124, for example.

As noted above, tracking module 340 may include one or more deep neuralnetworks, and may be configured to receive series of images 322/422 asinputs, and for each image return locations 352 including a list of 2Dskeletal landmarks corresponding to the pose included in the image,e.g., joint positions 228 a and 228 b. Tracking module 340 has beenpreviously trained over a large data set of body images, i.e., bodyimage dataset 336 a, as also noted above, but may be implemented so asto determine locations 352 including joint positions 228 a and 228 b ofsubject 124 based on each of images 122 a/222 a/122 b/222 b in anautomated process.

Tracking module 340 may be constrained to determine locations 352 basedon a hierarchical skeleton model in which 2D skeletal landmarks, such asjoint positions, are dependent on the position of one or more otherskeletal landmarks of subject 124, in contrast to a non-hierarchicalskeleton model in which individual skeletal landmarks are treated asindependent of one another. Determination of locations 352 may beperformed by software code 110/310, executed by hardware processor 104of computing platform 102, and using landmark detector 342/442 oftracking module 340, as discussed above by reference to FIG. 4.

Flowchart 780 continues with, for each image of series of images322/422, using tracking module 340 to infer joint angles 238 a/238 b/338of subject 124 based on locations 352 (action 783). It is noted thatalthough FIG. 2 depicts joint angles 238 a and 238 b as joint angles onhand(s) 126 a/126 b/226 of subject 124, that representation is merelyexemplary. More generally, joint angles 238 a/238 b/338 may correspondto the respective orientations of any relevant joint of body 125 ofsubject 124. Thus, in addition to, or as an alternative to hand joints,joint angles 238 a/238 b/338 may correspond to the orientations of hipjoints, leg joints, foot joints, shoulder joints, and arm joints ofsubject 124, as well as head, neck, and spine joints of subject 124.

Determination of joint angles 238 a/238 b/338 of 3D pose 354 may beperformed by software code 110/310, executed by hardware processor 104of computing platform 102, and using joint angle encoder 346/546 oftracking module 340 as discussed above by reference to FIG. 5.Furthermore, like action 782, action 783 may be performed as anautomated process.

Flowchart 780 continues with, for each image of series of images322/422, using tracking module 340 to reconstruct a 3D pose of subject124 based on locations 352 and joint angles 238 a/238 b/338, resultingin series of 3D poses 354 (action 783). Tracking module 340 may beconfigured to reconstruct a 3D pose for each image of series of images322/422 using inverse kinematics ANN 344/644 and decoder 348/648, asdiscussed above by reference to FIG. 6. That is to say an inversekinematic analytical or iterative process may be applied to 2D skeletallandmarks 228 a and 228 b included at locations 352 to determine a 3Dpose most closely corresponding to locations 352 and joint angles 238a/238 b/338. As discussed above, reconstruction of series of 3D posesbased 354 on locations 352 and joint angles 238 a/238 b/338 may beperformed by software code 110/310, executed by hardware processor 104of computing platform 102, and using fully connected inverse kinematicsANN 344/644 and decoder 348/648 of tracking module 340. Moreover, likeactions 782 and 783, action 784 may be performed as an automatedprocess.

In some implementations, flowchart 780 can conclude with outputtingtracking image 156/356 corresponding to the motion by subject 124 basedon series of 3D poses 354 by subject 124 (action 785). In someimplementations tracking image 156/356 may take the form of per frametracking image data corresponding respectively to the input frames ofseries of images 322/422. However, in other implementations, trackingimage 156/356 may include a synthesis of such per frame tracking imagedata to produce a substantially continuous replication of the motion bysubject 124.

Tracking image 156/356 corresponding to motion by subject 124 can beadvantageously utilized in a variety of applications. Examples of suchapplications include augmented reality (AR) applications, virtualreality (VR) applications, hand tracking based character animation, andextraction of motion by bipeds or quadrupeds from film footage, to namea few. Tracking image 156/356 may be output by software code 110/310,executed by hardware processor 104 of computing platform 102, and asnoted above, is based on series of 3D poses 354 reconstructed usingtracking module 340. In some implementations, hardware processor 104 mayfurther execute software code 110/310 to render tracking image 156/356on display 108.

Thus, the present application discloses a solution for performing poseestimation and body tracking using an ANN in a substantially automatedprocess. The pose estimation and body tracking solutions disclosed bythe present application make at least three significant contributions tothe conventional art. First, a novel and inventive landmark detector isused, that imposes anatomical constraints on the position of skeletallandmarks of a subject being tracked. Second, using a large dataset ofbody images, a Wasserstein autoencoder is trained to map joint angles ofa rigged hand or other body parts to a low dimensional latent space fromwhich plausible 3D poses can be reconstructed. Third, a fully connectedinverse kinematics ANN is introduced that learns to map positions ofskeletal landmarks in an image to the latent space of the Wassersteinautoencoder, thereby allowing accurate reconstruction of the pose of thesubject in 3D.

Consequently, the pose estimation and body tracking solution disclosedin the present application is more accurate than conventional approachesto pose estimation and body tracking using a color camera. In addition,the present solution enables use of a standard color camera for imagecapture, thereby advantageously avoiding any extra setup requirements.Furthermore, and in contrast to many conventional pose estimationtechniques that merely provide 2D joint locations, the present poseestimation and body tracking solution advantageously provides 3D posewith depth, and is able to do so under general lighting conditions. As aresult, the solution disclosed by the present application providesreliable, fast, accurate, and cost effective pose estimation and bodytracking.

From the above description it is manifest that various techniques can beused for implementing the concepts described in the present applicationwithout departing from the scope of those concepts. Moreover, while theconcepts have been described with specific reference to certainimplementations, a person of ordinary skill in the art would recognizethat changes can be made in form and detail without departing from thescope of those concepts. As such, the described implementations are tobe considered in all respects as illustrative and not restrictive. Itshould also be understood that the present application is not limited tothe particular implementations described herein, but manyrearrangements, modifications, and substitutions are possible withoutdeparting from the scope of the present disclosure.

What is claimed is:
 1. A pose estimation and body tracking systemcomprising: a computing platform including a hardware processor and asystem memory; a software code stored in the system memory, the softwarecode including a tracking module trained to track motions; the hardwareprocessor configured to execute the software code to: to receive aseries of images of a motion by a subject; for each image of the seriesof images, determine, using the tracking module, a plurality oflocations each corresponding respectively to a two-dimensional (2D)skeletal landmark of the subject based on constraints imposed byfeatures of a hierarchical skeleton model intersecting at each 2Dskeletal landmark; for each image of the series of images, infer, usingthe tracking module, a plurality of joint angles of the subject based onthe plurality of locations; for each image of the series of images,reconstruct, using the tracking module, a three-dimensional (3D) pose ofthe subject based on the plurality of locations and the plurality ofjoint angles, resulting in a series of 3D poses by the subject; andoutput a tracking image corresponding to the motion by the subject basedon the series of 3D poses by the subject.
 2. The pose estimation andbody tracking system of claim 1, wherein the hardware processor isfurther configured to execute the software code to render the trackingimage on a display.
 3. The pose estimation and body tracking system ofclaim 1, wherein the plurality of joint angles are represented asquaternions.
 4. The pose estimation and body tracking system of claim 1,wherein the tracking module is configured to determine the series of 3Dposes using a fully connected inverse kinematics artificial neuralnetwork (ANN).
 5. The pose estimation and body tracking system of claim1, wherein the subject comprises one of a human subject and a robot. 6.The pose estimation and body tracking system of claim 5, wherein themotion by the subject comprises a hand motion by the one of the humansubject and the robot.
 7. The pose estimation and body tracking systemof claim 1, wherein the series of images comprises a series of singlemonocular images.
 8. The pose estimation and body tracking system ofclaim 1, further comprising at least one camera configured to generatethe series of images, wherein a body of the subject is in motionrelative to the at least one camera.
 9. A method for use by a poseestimation and body tracking system including a computing platformhaving a hardware processor and a system memory storing a software codeincluding a tracking module trained to track motions, the methodcomprising: receiving, by the software code executed by the hardwareprocessor, a series of images of a motion by a subject; for each imageof the series of images, determining, by the software code executed bythe hardware processor and using the tracking module, a plurality oflocations each corresponding respectively to a two-dimensional (2D)skeletal landmark of the subject based on constraints imposed byfeatures of a hierarchical skeleton model intersecting at each 2Dskeletal landmark; for each image of the series of images, inferring, bythe software code executed by the hardware processor and using thetracking module, a plurality of joint angles of the subject based on theplurality of 2D locations; for each image of the series of images,reconstructing, by the software code executed by the hardware processorand using the tracking module, a three-dimensional (3D) pose of thesubject based on the plurality of locations and the plurality of jointangles, resulting in a series of 3D poses of the subject; andoutputting, by the software code executed by the hardware processor, atracking image corresponding to the motion by the subject based on theseries of 3D poses of the subject.
 10. The method of claim 9, furthercomprising rendering, by the software code executed by the hardwareprocessor, the tracking image on a display.
 11. The method of claim 9,wherein the plurality of joint angles are represented as quaternions.12. The method of claim 9, wherein the tracking module is configured todetermine the series of 3D poses using a fully connected inversekinematics artificial neural network (ANN).
 13. The method of claim 9,wherein the subject comprises one of a human subject and a robot. 14.The method of claim 13, wherein the motion by the subject comprises ahand motion by the one of the human subject and the robot.
 15. Themethod of claim 9, wherein the series of images comprises a series ofsingle monocular images.
 16. The method of claim 9, wherein the systemfurther comprises at least one camera configured to generate the seriesof images, and wherein a body of the subject is in motion relative tothe at least one camera.
 17. A method comprising: training an hourglassnetwork of a landmark detector of a tracking module to determine aplurality of locations corresponding respectively to a plurality oftwo-dimensional (2D) skeletal landmarks of a body image based onconstraints imposed by features of a hierarchical skeleton modelintersecting at each of the plurality of 2D skeletal landmarks; traininga joint angle encoder of the tracking module to map a plurality of jointangles of a rigged body part corresponding to the body image to a lowdimensional latent space from which a plurality of plausiblethree-dimensional (3D) poses of the body image can be reconstructed; andtraining an inverse kinematics artificial neural network (ANN) of thetracking module to map the plurality of locations correspondingrespectively to the 2D skeletal landmarks of the body image to the lowdimensional latent space of the joint angle encoder for accuratereconstruction of the pose of the body image in 3D.
 18. The method ofclaim 17, wherein the joint angle encoder is trained using a purelysynthetic dataset of skeletal landmark to joint angle correspondences.19. The method of claim 17, wherein the plurality of joint angles arerepresented as quaternions.
 20. The method of claim 17, wherein thejoint angle encoder is implemented as a Wasserstein autoencoder.